[SPARK-4163][Core][WebUI] Send the fetch failure message back to Web UI #3032

zsxwing · 2014-10-31T05:27:06Z

This is a PR to send the fetch failure message back to Web UI.
Before:

After (Please ignore the meaning of exception, I threw it in the code directly because it's hard to simulate a fetch failure):

SparkQA · 2014-10-31T05:35:14Z

Test build #22594 has started for PR 3032 at commit c261d23.

This patch merges cleanly.

SparkQA · 2014-10-31T06:25:15Z

Test build #22600 has started for PR 3032 at commit a3bca65.

This patch merges cleanly.

SparkQA · 2014-10-31T06:50:10Z

Test build #22594 has finished for PR 3032 at commit c261d23.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait FetchResult
- case class SuccessFetchResult(blockId: BlockId, size: Long, buf: ManagedBuffer)
- case class FailureFetchResult(blockId: BlockId, e: Throwable) extends FetchResult

AmplabJenkins · 2014-10-31T06:50:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22594/
Test PASSed.

SparkQA · 2014-10-31T07:36:34Z

Test build #22600 has finished for PR 3032 at commit a3bca65.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait FetchResult
- case class SuccessFetchResult(blockId: BlockId, size: Long, buf: ManagedBuffer)
- case class FailureFetchResult(blockId: BlockId, e: Throwable) extends FetchResult

AmplabJenkins · 2014-10-31T07:36:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22600/
Test PASSed.

SparkQA · 2014-11-01T04:39:41Z

Test build #22681 has started for PR 3032 at commit 0c07d1f.

This patch merges cleanly.

SparkQA · 2014-11-01T05:36:26Z

Test build #22681 has finished for PR 3032 at commit 0c07d1f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait FetchResult
- case class SuccessFetchResult(blockId: BlockId, size: Long, buf: ManagedBuffer)
- case class FailureFetchResult(blockId: BlockId, e: Throwable) extends FetchResult

AmplabJenkins · 2014-11-01T05:36:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22681/
Test FAILed.

aarondav · 2014-11-01T07:19:25Z

core/src/main/scala/org/apache/spark/shuffle/FetchFailedException.scala

  extends Exception {

  override def getMessage: String =
    "Fetch failed: %s %d %d %d".format(bmAddress, shuffleId, mapId, reduceId)

-  def toTaskEndReason: TaskEndReason = FetchFailed(bmAddress, shuffleId, mapId, reduceId)
+  def toTaskEndReason: TaskEndReason =


this may fit within 100ch on one line

aarondav · 2014-11-01T07:54:40Z

LGTM, only had superficial comments.

zsxwing · 2014-11-01T16:16:04Z

@aarondav thanks for reviewing it. Updated as per your comments.

SparkQA · 2014-11-01T16:19:48Z

Test build #22696 has started for PR 3032 at commit 62103fd.

This patch merges cleanly.

SparkQA · 2014-11-01T17:42:32Z

Test build #22696 has finished for PR 3032 at commit 62103fd.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- sealed trait FetchResult
- sealed case class SuccessFetchResult(blockId: BlockId, size: Long, buf: ManagedBuffer)
- sealed case class FailureFetchResult(blockId: BlockId, e: Throwable) extends FetchResult

AmplabJenkins · 2014-11-01T17:42:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22696/
Test PASSed.

aarondav · 2014-11-01T17:46:57Z

core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala

-  case class FetchResult(blockId: BlockId, size: Long, buf: ManagedBuffer) {
-    def failed: Boolean = size == -1
-    if (failed) assert(buf == null) else assert(buf != null)
+  sealed case class SuccessFetchResult(blockId: BlockId, size: Long, buf: ManagedBuffer)


ah, "sealed" is usually only applied to traits, as case classes themselves are not supposed to be extended.

Already replaced sealed with private[storage]. Thank you.

zsxwing · 2014-11-02T06:59:48Z

How about now?

SparkQA · 2014-11-02T07:04:51Z

Test build #22753 has started for PR 3032 at commit 316767d.

This patch merges cleanly.

AmplabJenkins · 2014-11-02T07:12:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22752/
Test FAILed.

SparkQA · 2014-11-02T09:00:01Z

Test build #22757 has started for PR 3032 at commit 4e946f7.

This patch merges cleanly.

SparkQA · 2014-11-02T09:25:35Z

Test build #22753 has finished for PR 3032 at commit 316767d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-02T09:25:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22753/
Test PASSed.

SparkQA · 2014-11-02T10:00:02Z

Test build #22757 timed out for PR 3032 at commit 4e946f7 after a configured wait of 120m.

AmplabJenkins · 2014-11-02T10:00:05Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22757/
Test FAILed.

JoshRosen · 2014-11-02T18:51:51Z

Jenkins, retest this please.

SparkQA · 2014-11-02T18:55:06Z

Test build #22764 has started for PR 3032 at commit 4e946f7.

This patch merges cleanly.

SparkQA · 2014-11-02T20:55:06Z

Test build #22764 timed out for PR 3032 at commit 4e946f7 after a configured wait of 120m.

AmplabJenkins · 2014-11-02T20:55:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22764/
Test FAILed.

aarondav · 2014-11-02T21:45:10Z

core/src/main/scala/org/apache/spark/shuffle/hash/BlockStoreShuffleFetcher.scala

          blockId match {
            case ShuffleBlockId(shufId, mapId, _) =>
              val address = statuses(mapId.toInt)._1
-              throw new FetchFailedException(address, shufId.toInt, mapId.toInt, reduceId)
+              throw new FetchFailedException(address, shufId.toInt, mapId.toInt, reduceId, e)


I tested this patch out locally, and it appears that, sadly, the cause is actually not propagated correctly back to the driver. I think we can revert this to the prior Utils.exceptionString() way, that looks good in the UI.

I will revert to the Utils.exceptionString() way.

Looks Utils.exceptionString() does not handle the nested exceptions like this. It only outputs the outermost exception info. Is it intentional?

That is almost certainly not intentional. I think the method was introduced rather recently, and does not have many users right now. It should be fixed.

I will try to fix it in another PR.

zsxwing · 2014-11-03T05:19:21Z

New screenshots for current codes (I threw the exception in onBlockFetchSuccess directly to make up an error)

Stages page:

Detail for stage:

SparkQA · 2014-11-03T05:19:45Z

Test build #22800 has started for PR 3032 at commit f7e1faf.

This patch merges cleanly.

aarondav · 2014-11-03T05:30:13Z

LGTM, will merge as soon as Jenkins passes.

SparkQA · 2014-11-03T06:46:50Z

Test build #22800 has finished for PR 3032 at commit f7e1faf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-03T06:46:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22800/
Test PASSed.

aarondav · 2014-11-03T07:20:05Z

Merging into master. Thanks for this!

Send the fetch failure message back to Web UI

a3bca65

zsxwing changed the title ~~[SPARK-4163][Core][Web UI] Send the fetch failure message back to Web UI~~ [SPARK-4163][Core][WebUI] Send the fetch failure message back to Web UI Oct 31, 2014

JoshRosen mentioned this pull request Oct 31, 2014

[SPARK-4166][Core][WebUI] Display the executor ID in the Web UI when ExecutorLostFailure happens #3033

Closed

Backward-compatible support

0c07d1f

aarondav referenced this pull request Nov 1, 2014

Do not unregister executor outputs unduly

fd3928b

aarondav reviewed Nov 1, 2014
View reviewed changes

Update as per review

62103fd

aarondav reviewed Nov 1, 2014
View reviewed changes

zsxwing added 2 commits November 2, 2014 14:52

Set e as the cause of FetchFailedException

d51b0b6

Add private[storage] to FetchResult

316767d

Add e as the cause of SparkException

4e946f7

aarondav reviewed Nov 2, 2014
View reviewed changes

Discard changes for FetchFailedException and minor modification

f7e1faf

asfgit closed this in 76386e1 Nov 3, 2014

zsxwing deleted the SPARK-4163 branch November 3, 2014 07:50

zsxwing mentioned this pull request Nov 4, 2014

[SPARK-4204][Core][WebUI] Change Utils.exceptionString to contain the inner exceptions and make the error information in Web UI more friendly #3073

Closed

[SPARK-4163][Core][WebUI] Send the fetch failure message back to Web UI #3032

[SPARK-4163][Core][WebUI] Send the fetch failure message back to Web UI #3032

Uh oh!

Conversation

zsxwing commented Oct 31, 2014

Uh oh!

SparkQA commented Oct 31, 2014

Uh oh!

SparkQA commented Oct 31, 2014

Uh oh!

SparkQA commented Oct 31, 2014

Uh oh!

AmplabJenkins commented Oct 31, 2014

Uh oh!

SparkQA commented Oct 31, 2014

Uh oh!

AmplabJenkins commented Oct 31, 2014

Uh oh!

SparkQA commented Nov 1, 2014

Uh oh!

SparkQA commented Nov 1, 2014

Uh oh!

AmplabJenkins commented Nov 1, 2014

Uh oh!

aarondav Nov 1, 2014

Choose a reason for hiding this comment

Uh oh!

aarondav commented Nov 1, 2014

Uh oh!

zsxwing commented Nov 1, 2014

Uh oh!

SparkQA commented Nov 1, 2014

Uh oh!

SparkQA commented Nov 1, 2014

Uh oh!

AmplabJenkins commented Nov 1, 2014

Uh oh!

aarondav Nov 1, 2014

Choose a reason for hiding this comment

Uh oh!

zsxwing Nov 2, 2014

Choose a reason for hiding this comment

Uh oh!

zsxwing commented Nov 2, 2014

Uh oh!

SparkQA commented Nov 2, 2014

Uh oh!

AmplabJenkins commented Nov 2, 2014

Uh oh!

SparkQA commented Nov 2, 2014

Uh oh!

SparkQA commented Nov 2, 2014

Uh oh!

AmplabJenkins commented Nov 2, 2014

Uh oh!

SparkQA commented Nov 2, 2014

Uh oh!

AmplabJenkins commented Nov 2, 2014

Uh oh!

JoshRosen commented Nov 2, 2014

Uh oh!

SparkQA commented Nov 2, 2014

Uh oh!

SparkQA commented Nov 2, 2014

Uh oh!

AmplabJenkins commented Nov 2, 2014

Uh oh!

aarondav Nov 2, 2014

Choose a reason for hiding this comment

Uh oh!

zsxwing Nov 3, 2014

Choose a reason for hiding this comment

Uh oh!

aarondav Nov 3, 2014

Choose a reason for hiding this comment

Uh oh!

zsxwing Nov 3, 2014

Choose a reason for hiding this comment

Uh oh!

zsxwing commented Nov 3, 2014